Fix validation result status transitions - #195

Pull request overview

This PR fixes the “validation result ingestion” pipeline so that alert-parser can observe validation success/failure alerts, and updates the resulting node status transitions to prevent reclassification loops after validation failures.

Changes:

• Emit RecoverValidatedNodes and CordonValidationFailedNodes alerts with severity: error so alert-parser’s existing query can consume them.
• Change failed validation state transition from validating -> cordoned to validating -> triaged_unknown.
• Extend alert-parser unit coverage for validation success (available_nodata) and failure (triaged_unknown) transitions.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

      severity: "error",
      node_name: node,
    },
    annotations: {
      summary: `The node ${node} has been validated and be uncordoned.`,

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Thanks for the fix — the core direction makes sense, especially the transition from validating to triaged_unknown and the matching test updates. I have a few follow-up suggestions:\n\n1) Severity semantics for RecoverValidatedNodes\nRecoverValidatedNodes represents a successful validation/uncordon path, but this PR changes its severity from info to error in alert.js. That may create noisy/error-classified alerts for success events.\n\nSuggestion: keep RecoverValidatedNodes as info, and make alert-parser robust to both info/error for this alertname (or filter by alertname instead of strict severity where appropriate).\n\n2) Timestamp consistency for state transition\nFor CordonValidationFailedNodes, transition time uses the alert event timestamp, but RecoverValidatedNodes still uses the polling timestamp. In delayed ingestion scenarios, this can cause status timeline skew.\n\nSuggestion: for RecoverValidatedNodes, also use the alert event timestamp (similar to validation failure branch) when calling update_status_action.\n\n3) Missing contract test around alert payload\nThe parser behavior depends on alert payload fields (alertname/severity/status), but there is no focused test around job-status-change-notification alert payload generation for these validation alerts.\n\nSuggestion: add a small unit test for alert.js to lock down alertname + severity + status for cordon/uncordon payloads and prevent regressions.\n\nOverall: logic fix looks good; with the above adjustments the change should be safer and easier to maintain.